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PREFACE 


One  of  the  early  tasks  undertaken  by  the  RAND/UCLA  Center  for 
Health  Care  Financing  Policy  Research  was  the  preparation  of 
instructional  materials  to  be  included  in  the  standard  application  kits 
distributed  by  the  Office  of  Demonstrations  and  Evaluations,  Office  of 
Research  and  Demonstrations,  Health  Care  Financing  Administration 
(ODE/ORD/HCFA)  to  persons,  government  agencies,  and  institutions  wishing 
to  apply  for  funds  or  waivers  to  carry  out  a  demonstration  project.  For 
one  of  these  tasks,  HCFA  asked  that  the  center  develop  a  short,  informal 
booklet  that  would  explain  the  important  attributes  of  a  successful 
grant  proposal,  with  specific  attention  to  the  "Project  Narrative" 
portion  of  the  application.    The  audience  was  assumed  to  be  relatively 
unfamiliar  with  the  basic  principles  of  research,  HCFA's  program  of 
demonstrations,  and  the  general  process  of  peer  review  of  demonstration 
applications . 

Although  the  resulting  booklet  is  tailored  for  a  HCFA  proposal,  its 
underlying  themes  and  suggestions  should  be  of  interest  to  any 
individuals  or  institutions  that  wish  to  submit  applications  to  the 
various  governmental  and  private  agencies  and  foundations  that  fund 
health  services  and  health  policy  research.     The  project  and  the  Center 
are  supported  through  Cooperative  Research  Agreement  18-C-98489/9-02. 
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SUMMARY 


Many  individuals  and  institutions  have  good  ideas  for  new  ways  to 
provide  high  quality  health  care,  while  containing  costs,  within  two 
major  publicly  funded  programs,  Medicare  and  Medicaid.     To  obtain 
funding  to  demonstrate  these  ideas,  people  like  you  at  institutions  like 
yours  may  elect  to  submit  an  application  to  the  Office  of  Research  and 
Demonstrations  of  the  Health  Care  Financing  Administration  (HCFA) .  To 
be  successful  in  this  search  for  financing  or  a  waiver  to  conduct  a 
demonstration  project,  you  must  develop  a  proposal  that  describes: 

•  What  you  propose  to  do; 

•  Why  you  propose  to  do  it  in  the  manner  described; 

•  Why  the  enterprise  is  worthwhile,  in  its  own  right  and  to  HCFA; 
and 

•  What  new  contributions  your  project  offers  (and  how  it  is 
related  to  past  or  current  work  by  you  or  others). 

In  developing  a  demonstration  proposal,  and  especially  the  section 
known  as  the  "Project  Narrative"  that  describes  the  actual  research  you 
intend  to  carry  out,  it  is  important  to  keep  in  mind  the  main  audience 
for  which  you  are  writing.     This  is  the  peer  review  panel,  sometimes 
known  as  a  study  section,  that  reads,  evaluates,  and  passes  judgment  on 
your  proposed  study.     Your  peer  reviewers  will  look  to  see  if  you  have 
identified  the  important  effects  or  outcomes  to  study  and  if  you  have 
designed  your  study  in  a  way  that  will  let  you  detect  those  outcomes  if 
they  occur  and  determine  the  correct  causal  factors.     In  this  brief 
booklet,  we  attempt  to  provide  some  guidance  and  reminders  about  what  is 
or  is  not  found  in  a  successful  proposal. 

We  describe  and  discuss  the  major  elements  of  the  project  narrative 
section  of  a  demonstration  grant  proposal,  namely:     Project  Title  and 
Objectives,  Background  and  Importance,  Research  Questions  and  Methods, 
Evaluation  and  Analysis  Plan,  Work  Plan,  Project  Staff,  and 
Implementation  Potential. 
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We  concentrate  on  several  topics  that  should  receive  special 
attention  during  your  proposal-writing.     These  include  hypotheses  to  be 
tested  (or  that  might  be  tested  in  a  follow-on  study);  study  design 
issues  such  as  experimental  variables,  population  and  samples  to  be 
studied;  and  plans  and  problems  for  data  collection.    We  also  offer  some 
suggestions  about  the  level  of  detail  in  your  analysis  plan  and  the 
importance  of  specifying  any  analytic  pitfalls  you  can  foresee.  For 
instance,  we  argue  that  you  need  to  make  sure  that  you  have  no  loose 
ends  as  regards  the  data  you  plan  to  collect  and  how  you  plan  to  use 
them  in  your  final  analyses.     Finally,  we  have  appended  a  highly 
selective  bibliography  on  study  design  and  analysis  topics  and  a 
narrative  glossary  of  scientific  and  statistical  terms  and  synonyms  that 
may  be  helpful. 
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I.  PURPOSE  OF  HCFA  DEMONSTRATION  PROJECTS 

The  Office  of  Research  and  Demonstrations  (ORD)  of  the  Health  Care 
Financing  Administration  (HCFA)  directs  several  hundred  research, 
evaluation,  and  demonstration  projects.     These  projects  are  used  to  test 
new  ways  to  provide  high  quality  health  care,  while  containing  costs, 
within  two  major  health  programs,  Medicare  and  Medicaid.     Such  projects 
often  provide  a  basis  for  decisions  about  critical  health  policy  issues, 
and  they  frequently  advance  the  state  of  knowledge  about  measuring 
health  care  practices  and  behaviors  of  providers  and  program 
beneficiaries.     This  booklet  is  intended  as  an  aid  to  individuals  and 
institutions  who  wish  to  secure  HCFA  funding  for  demonstration  projects. 

II.  PURPOSE,  SCOPE,  AND  ORGANIZATION  OF  THIS  BOOKLET 

You  have  in  mind  a  good  idea  "to  demonstrate"  and  you  believe  that 
it  corresponds  to  HCFA's  concerns  and  research  priorities.     That  is  a 
long  way,  however,  from  a  successful  proposal  that  will  gain  approval 
during  the  review  process  and,  ultimately,  gain  funding  or  a  waiver  from 
HCFA.     This  booklet  attempts  to  provide  some  guidance  and  reminders 
about  what  is  and  is  not  found  in  a  successful  proposal. 

The  Office  of  Demonstrations  and  Evaluations  (ODE)  within  ORD 
funds,  manages,  and  evaluates  projects  or  studies  that  assess  new  and 
innovative  ways  of  delivering  and  financing  Medicare's  and  Medicaid's 
services.     ODE  defines  three  types  of  demonstrations,  which  are  briefly 
described  in  Table  1.     Briefly,  demonstration  studies  can  be  controlled 
experiments  in  which  the  investigator  manipulates  treatment  variables 
and  randomly  assigns  participants  to  groups,  observational  studies  in 
which  the  investigator  manipulates  treatment  variables  but  cannot 
control  who  goes  into  which  groups,  or  descriptive  studies  in  which  the 
investigator  examines  new  or  existing  programs  or  groups  to  see  how  well 
they  operate  or  how  feasible  they  are  to  expand. 

You  are  expected  to  designate  your  proposal  as  belonging  to  one  of 
these  categories,  and  this  booklet  provides  proposal-writing  suggestions 
applicable  to  all  three.     However,  many  of  the  Category  I  proposals, 
generically  considered  experiments  (randomized  controlled  trials),  will 
be  written  by  researchers  with  considerable  experience,  so  much  of  this 
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Table  1 

SUMMARY  OF  CHARACTERISTICS  OF  HCFA  DEMONSTRATION  CATEGORIES 


Type  of  HCFA  Demonstration 

Characteristic 


or  Requirement 

Category  I 

Category  II 

Category  III 

Type  of 

Experimental 

Quasi- 

Descriptive 

Research  Design 

(.controlled 

experimental 

experiment) 

(observational 

study) 

Major 

Investigator 

Investigator 

No  artificial 

Characteristics 

manipulates 

manipulates 

manipulation 

and  Subtypes 

treatment 

treatment 

of  variables 

variables 

variables 

Randomizat  ion 

No  randomization 

No  randomization 

of  study 

of  study 

of  study 

participants 

participants 

participants 

into 

experimental 

Use  of  comparison 

Use  of  comparison 

and  control 

group (s)  or 

group (s)  or 

group (s) 

before/after 

before/after 

comparison  of 

comparisons  of 

same  study 

same  study 

participants 

participants 

HCFA 

Research  and 

Separate 

Must  discuss 

Evaluation  Plan 

evaluation 

evaluation 

evaluation  issues 

Requirements 

plan  are  the 

plan 

and  how  to 

same 

required 

approach  them 

Applicants 

Applicants 

HLrA  decides 

conduct 

must  include 

whether  applicant 

evaluation  or 

evaluation 

or  HCiA  is 

designate 

component 

responsible  for 

evaluator 

(HCFA  may  still 

evaluation 

do  an  independent 

evaluation) 

Relative  Importance 

Critical 

Quite 

Important 

of  Design  Issues 

important 

(programmat  ic 

and  feasibility 

issues  dominate) 

Relative  Importance 

Critical 

Critical 

Critical 

of  Evaluation  Issues 

(but  builds 

directly 

from  design) 
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booklet  is  addressed  to  persons  or  agencies  who  may  have  less  expertise 
in  this  area  and  who  may  be  more  likely  to  propose  Category  II  or  III 
studies . 

We  focus  here  on  the  description  you  will  give  of  your  research 
project,  especially  in  the  section  of  the  proposal  called  "Project 
Narrative."    We  are  not  concerned  with  how  one  estimates  or  prepares  a 
budget  or  develops  waiver  cost  estimates  for  projects  that  need  them.1 
The  remainder  of  the  booklet  treats  the  purpose  of  your  proposal, 
important  parts  of  your  proposal,  and  key  elements  of  the  project 
narrative.     In  addition,  we  have  appended  both  a  highly  selective 
bibliography  of  reference  publications  that  you  may  find  helpful  and  a 
glossary  that  explains  more  fully  some  of  the  terms  contained  in  this 
booklet . 

III.  PURPOSE  OF  A  PROPOSAL:    COMMUNICATE  TO  REVIEWERS 

The  first  and  most  important  audience  you  need  to  reach  is  the  peer 
review  panel,  which  will  read,  evaluate,  and  pass  judgment  on  your 
proposed  study.     Thus,  the  basic  purpose  of  your  proposal  is  to 
communicate  your  ideas  to  this  panel,  thoroughly  yet  crisply.     If  you 
cannot  successfully  outline  your  objectives,  explain  your  study  methods, 
or  argue  for  the  importance  of  your  project,  your  proposal  will  quite 
probably  be  disapproved  on  scientific  and  technical  grounds. 

Announcements  of  the  demonstration  program,  typically  appearing  in 
the  Federal  Register ,  give  due  dates  for  applications,  and  peer  review 
panels  are  usually  convened  about  a  month  or  two  after  these  closing 
dates.     The  panels  consist  of  experts--"peers"--f rom  HCFA,  other  federal 

1 Advice  on  completing  the  budget  forms  can  be  found  in  the  HCFA 
brochure  entitled  "Research  &  Demonstrations  Grant  Application  Guide," 
which  is  included  in  this  grant  application  kit;  instructions  about 
waiver  cost  estimates  are  given  in  the  HCFA  document  "Guidelines  for  the 
Preparation  of  Demonstration  Project  Waiver  Cost  Estimates,"  OMB 
#09380408,  which  is  also  enclosed  in  the  application  kit.     You  should 
take  care  to  fill  out  all  required  forms  completely. 
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health  agencies,  and  (mainly)  from  academic  and  research  institutions 
around  the  country.     These  peers  represent  a  range  of  disciplines  from 
economics  and  statistics  to  psychology  and  sociology  to  medicine  and 
health  policy.     Always  assume  that  at  least  one  member  of  a  review  panel 
is  knowledgeable  about  the  topics  you  want  to  investigate  and  the 
methods  you  propose  to  use  in  your  investigation;  panel  members  who  have 
this  knowledge  may  well  be  assigned  as  the  primary  reviewers  of  your 
proposal.     But  remember  also  that  you  must  communicate  with  the  entire 
review  panel  and  that  many  panel  members  may  not  have  specialized 
knowledge  in  your  particular  area. 

Peer  review  panels,  sometimes  known  as  study  sections,  receive 
applications  to  be  reviewed  in  advance  and  then  meet  for  (usually)  a  day 
or  two;  in  that  time,  they  may  discuss,  critique,  and  vote  on  10  to  25 
or  more  proposals.     This  means  that,  often,  proposals  are  read  and 
reviews  written  under  great  time  constraints.     Thus,  all  that  you  can  do 
to  make  your  proposal  as  clear  and  concise  as  possible,  consistent  with 
telling  the  full  story  about  your  intended  project,  will  pay  immense 
rewards . 

In  particular,  you  must  find  balance:     Communicate  enough,  but  not 
too  much.     Reviewers  should  be  able  to  understand  all  of  the  following: 

•  What  you  propose  to  do; 

•  Why  you  propose  to  do  it  in  the  manner  described', 

•  Why  the  enterprise  is  worthwhile ,  in  its  own  right  and  to  HCFA; 
and 

•  What  new  contributions  your  project  offers  (and  how  it  is 
related  to  past  or  current  work  by  you  or  others). 

Reviewers  should  not  be  confronted  with  extraneous  material, 
excessively  long  literature  reviews,  or  unsubstantiated  claims  about  the 
project's  relevance  or  importance.     Within  some  reasonable  length,  you 
want  to  convince  the  reviewer  that  your  topic  is  worth  investigation  and 
that  you  can  handle  the  task. 

A  completed  demonstration  application  will  have  five  main  parts: 
the  "Front  Sheet,"  the  "Project  Approval  Information  Sheet,"  the  "Budget 
Information  and  Justification,"  the  "Project  Narrative,"  and  the  "Waiver 
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Cost  Estimate."     (See  your  application  kit  for  these  forms.)    All  are 
eventually  important  in  final  funding  decisions.     The  most  critical  is 
the  Project  Narrative ,  because  it  is  the  heart  of  your  proposal  and  as 
such  is  given  the  most  scrutiny  by  the  review  panel. 

Over  the  years,  conventions  have  emerged  about  the  structure  of 
research  applications,  including  standard  outlines.     The  Project 
Narrative  is  no  exception.     Although  the  outline  suggested  in  Table  2  is 
not  an  absolute  requirement,  it  is  a  commonly  used  guide  for  HCFA 
proposals.     Thus,  it  is  used  here  as  the  format  for  discussing  the  major 
points  about  preparing  a  good  proposal.     If  you  believe  you  have  good 
reason  to  depart  from  this  outline,  do  so.     Remember,  however,  that  your 
reviewers  will  expect  something  much  like  this  structure,  and  you 
lighten  their  task  when  you  follow  it. 

IV.    PROJECT  NARRATIVE:    KEY  ELEMENTS 

A.  Project  Title  and  Objectives 

Be  succinct  in  titling  your  project,  and  be  accurate.     Find  the  key 
words,  phrases,  or  descriptors  that  will  highlight  the  population  of 
interest,  the  medical  problems  of  concern,  and  the  health  policy  issues 
of  importance,  and  then  stop. 

Your  objectives  should  pinpoint  what  you  plan  to  do  and  what  you 
expect  to  achieve.     They  should  be  relatively  few  in  number  and  listed 
in  approximate  order  of  priority  or  importance.     Remember  that  what  you 
state  as  your  objectives  sets  the  framework  and  the  tone  for  judging  the 
rest  of  the  proposal.     Underscore  the  major  elements  of  your  work  that 
you  believe  are  achievable:     Do  not  promise  to  study  the  world  or  to 
answer  all  the  crucial  questions  in  the  area. 

B.  Background  and  Importance 
B.I.    Background  to  the  Project 

This  is  in  all  likelihood  where  you  will  put  your  1 iterature 
review.     It  need  not  be  lengthy,  but  it  should  be  reasonably 
comprehensive  and  up-to-date.     Basically,  your  task  is  to  highlight, 
succinctly,  where  the  gaps  in  knowledge  or  practice  are  that  your 
project  will  help  correct.     You  must  show  that  you  understand  the 
important  studies  that  form  the  foundation  for  your  proposal  and 
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Table  2 

SAMPLE  PROJECT  NARRATIVE  OUTLINE  FOR  HCFA 
DEMONSTRATION/EVALUATION  GRANT  PROPOSALS 


(Part      I.     Front  Sheet) 

(Part    II.     Project  Approval  Information  Sheet) 
(Part  III.     Budget  Information  and  Justification) 

Part    IV.     Project  Narrative 

Table  of  Contents  Page 

Executive  Summary 

A.  Project  Title  and  Objectives 

B .  Background  and  Importance 

C.  Research  Questions  and  Methods 

1.  Hypotheses 

2.  Study  Design 

D.  Evaluation  and  Analysis  Plan 

E.  Work  Plan 

1.  Description  of  Tasks 

2.  Time  Schedule 

3.  Level  of  Effort  of  Personnel 

F.  Project  Staff 

1.  Qualifications  of  Key  Staff 

2.  Organizational  Chart 

G.  Implementation  Potential 
References 

Appendices 
(Part      V.     Waiver  Cost  Estimate) 


indicate  how  your  project  will  go  beyond  them.     You  are  not  expected  to 
review  all  the  relevant  literature  in  great  detail;  if  you  are 
conversant  with  other  bibliographies  or  literature  reviews,  go  ahead  and 
cite  them. 
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Again,  assume  that  at  least  one  reviewer  will  know  this  literature 
well;  he  or  she  will  speak  up  during  the  panel  meeting  if  the  proposal 
shows  no  familiarity  with  existing  work.     If  there  is  no  literature  or 
body  of  knowledge  in  the  area  you  want  to  study,  say  so  up  front --but  be 
sure  you  are  right.     Rarely  does  a  project  start  de  novo,  so  to  be  safe 
you  may  still  be  better  off  briefly  considering  the  research  closest  to 
your  proposed  work. 

For  demonstration  proposals,  it  is  important  to  show  familiarity 
with  HCFA-sponsored  work.     Your  literature  review  will  presumably  pick 
up  relevant  published  articles  or  reports.     For  on-going  projects,  you 
might  double-check  the  HCFA  publication  called  Health  Care  Financing 
Status  Report,  which  appears  annually.     Indicate  where  your  proposed 
work  builds  on  earlier  or  current  projects  or  addresses  new  problems  not 
yet  investigated  through  HCFA  funding.     This  often  provides  a  lead- 
in  to  the  next  subsection,  on  "importance  of  the  Project." 

Finally,  there  are  two  other  scholarly  points:     Cite  the  references 
completely  and  correctly,  and  be  sure  everything  has  a  citation  in  the 
reference  list.     You  can  be  sure  that  if  you  omit  something  from  the 
list,  that  will  be  the  reference  your  reviewers  will  want  to  know  about. 
Of  course,  you  will  not  cite  anything  that  you  have  not  used  in  the 
literature  review  or  in  later  parts  of  the  proposal. 

B.2.    Importance  of  the  Project 

You  need  to  make  two  main  points  here:     the  significance  of  the 
question  or  issue  you  are  proposing  to  investigate  and  the  significance 
of  your  particular  project.     As  to  the  former,  HCFA  occasionally 
publishes  a  grant  solicitation  in  the  Federal  Register .     Recent  ones 
(e.g.,  that  of  January  30,  1985)  highlight  priority  areas  for 
HCFA-sponsored  research.     If  your  topic  fits  in  one  of  the  areas 
specifically  mentioned  in  such  a  solicitation,  say  so,  because  proposals 
in  these  areas  receive  priority  for  funding.     This  also  assures  that 
your  proposal  is  reviewed  by  the  appropriate  review  panel. 

As  to  the  significance  of  your  proposed  work:     This  is  the  place  to 
make  as  strong  a  case  as  possible  for  the  importance  of  your  particular 
project!     It  may  add  to  the  general  body  of  knowledge  about  a  problem; 
it  may  expand  the  possible  ways  to  organize  and  deliver  health  services 
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to  meet  a  particular  human  need;  it  may  do  both.     The  point  is  to 
marshall  a  credible,  straightforward  argument  for  the  important 
contributions  your  work  will  make. 

C.    Research  Questions  and  Methods 

Together  with  the  subsection  "Evaluation  and  Analysis  Plan,"  this 
is  the  heart  of  the  Project  Narrative.     These  sections  will  be  read  more 
carefully  than  any  others,  because  ultimately  this  is  what  HCFA  is 
buying . 

Hence,  your  peer  reviewers  will  look  to  see  if  you  have 

•  Identified  the  important  effects  or  outcomes  to  study;  and 

•  Designed  your  study  in  a  way  that  will  let  you  detect  those 
effects  if  they  occur  and  determine  the  correct  causal  factors. 

C.I.  Hypotheses 

If  you  have  hypotheses  to  test,  state  them  explicitly.     Category  I 
and  II  demonstrations  probably  will  have  them.     Category  III 
demonstrations  may  not  have  precise,  testable  hypotheses,  as  they  are 
usually  descriptive  studies,  but  you  should  at  least  allude  to  the 
issues  that  prompted  you  to  undertake  the  project.     In  some  cases,  a 
Category  III  study  is  seen  as  a  precursor  to  a  larger,  more  controlled 
experiment.     In  this  case,  it  may  be  useful  to  indicate  what  hypotheses 
would  be  tested  in  a  larger  study  in  the  event  your  first  study 
indicates  that  a  followup  would  be  worthwhile. 

C.2.    Study  Design 

Your  basic  objective  here  is  to  describe  how  your  demonstration 
project  will  operate.     In  part,  you  could  try  to  discuss  how  your 
demonstration  activities  are  similar  to  or  different  from  what  occurs 
now  in  the  programs,  Medicare  or  Medicaid,  for  instance,  that  you  are 
trying  to  study. 

The  research  methods  will  come  under  close  scrutiny  in  any  review. 
They  may  be  more  critical,  the  more  like  a  Category  I  study  your  project 
is,  but  in  all  cases  they  should  be  thought  out.     It  is  crucial  that  the 
timing  and  sequence  of  your  project  be  clear  in  the  reviewers'  minds; 
you  may  find  that  inserting  a  simple  descriptive  diagram  at  this  point 
that  makes  the  timeline  clear  will  prove  very  helpful. 
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Illustrative  questions  that  should  be  addressed  directly  in  the 
proposal  are  briefly  noted  below,  but  they  do  not  necessarily  exhaust 
the  important  dimensions  to  study  design  that  may  be  pertinent  in  your 
particular  case. 

Treatment  or  experimental  variables.     In  some  ways,  the  key  part 
of  your  study  will  be  the  activities  or  services  you  undertake  to  study- 
that  is,  the  treatment  or  experimental  actions  that  are  under  your 
control.     This  might  be  a  single  line  of  action,  such  as  providing  a 
particular  nursing  service  on  an  ambulatory  basis;  or,  it  may  be  a  range 
of  related  activities  of  varying  intensity  (such  as  providing  only 
"usual  medical  care,"  "usual  care  plus  homemaker  services,"  or  "usual 
care  plus  homemaker  and  visiting  nurse  services").     In  any  event,  you 
must  make  sure  you  have  described  these  as  fully  as  possible. 

Population  to  be  studied.    What  is  the  population  from  which  your 
study's  participants  and  subjects  will  be  drawn?     Is  it  truly 
appropriate  to  the  topic  you  want  to  examine  and  the  program  you  want  to 
demonstrate?    What  will  be  your  unit  of  observation?    For  instance,  are 
you  studying  individuals,  organizations,  or  communities? 

Sample  to  be  used.    Rarely  will  a  demonstration  project  study 
everybody  in  a  population  or  community;  typically,  you  will  use  a  sample 
drawn  from  a  larger  group.     In  either  case,  you  should  explain  as  much 
as  you  can  about  how  you  will  determine  the  size  of  your  sample  and  how 
you  will  select  it,  how  you  will  assign  your  study  participants  to 
different  groups,  and  how  you  will  deal  with  sample  loss.     It  is  often 
helpful  to  present  a  summary  table  giving  information  about  the  initial 
sample  size,  the  likely  number  of  subjects  lost  from  the  sample  for 
various  reasons,  and  the  target  numbers  of  subjects  in  each  group. 

Regarding  sample  size  and  generalizability :     How  many  people  will 
you  be  studying?    How  representative  are  they  of  the  population  of 
interest  (e.g.,  Medicaid  or  Medicare  beneficiaries)?    How  many  will  be 
in  your  experimental  or  treatment  groups  and  how  many  in  the  control  or 
comparison  groups?    Will  these  numbers  be  sufficient  to  let  you  detect 
differences  of  a  magnitude  relevant  to  health  policy  decisions?  These 
questions  relate  to  the  important  issue  of  the  precision  or  power  of 
your  study  and  the  strength  of  its  eventual  conclusions,  so  you  should 
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indicate  here  (or  in  an  appendix)  whatever  power  calculations  you  might 
have  done  to  justify  your  sample  sizes.    Will  your  sample  let  you 
generalize  accurately  to  larger  populations? 

If  this  is  a  true  experiment  (Category  I),  how  will  you  randomly 
assign  participants  to  groups?    Do  you  face  any  ethical  or  practical 
problems  in  this  assignment  process?     If  this  is  a  quasi -experiment 
(Category  II  and  possibly  Category  III),  how  will  you  match  experimental 
to  comparison  groups  or  otherwise  find  a  suitable  external  comparison? 
In  this  latter  case,  you  should  take  on  directly  the  question  of  why  you 
rejected  a  full  randomized  design. 

As  for  sample  loss,  what  will  you  do  if  people  drop  out  of  your 
study?    That  is,  what  if  some  individuals  leave  the  area,  become 
disenchanted  with  your  project,  become  too  ill  to  continue  to 
participate,  or  die?    Do  you  have  some  idea  how  large  these  losses  might 
be?    Have  you  built  enough  subjects  into  your  total  sample  to  withstand 
such  sample  losses?    What  provisions  have  you  made  for  dealing  with  bias 
arising  from  these  losses  (systematic  differences  between  the  subjects 
who  drop  out  and  those  who  remain)? 

Data  collection  plans.    Describing  fully  your  plans  for  gathering 
information  is  critical.     You  should  provide  as  much  detail  as  you  can 
at  this  point  to  answer  the  following:    What  pieces  of  information  are 
you  planning  to  collect?    Precisely  from  whom?    How  often?    By  what 
techniques?    Are  there  alternative  data  collection  methods  or  sources  of 
information  that  you  have  considered  but  rejected?     If  so,  explain  why, 
especially  if  the  ones  you  dismissed  might  be  less  costly. 

Uppermost  in  the  reviewers'  minds  may  be  the  question  of  how  each 
piece  of  information  relates  to  the  hypotheses  you  are  testing  or  the 
program  you  are  trying  to  demonstrate.     Think  of  it  as  a  chain  of 
reasoning  that  must  be  internally  consistent --an  unbroken  set  of  links, 
so  to  speak.     These  links  are  critical,  and  you  can  make  the  points  best 
if  you  try  to  accomplish  the  following  here: 

•      Give  a  good,  specific  description  of  the  match  between  what  you 
say  you  are  investigating  and  the  particular  data  you  plan  to 
collect . 
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•  Clarify  what  your  dependent  (or  response)  variables  are,  what 
your  independent  (or  treatment  or  explanatory)  variables  are, 
and  what  factors  you  may  need  to  measure  or  account  for  because 
they  might  otherwise  confound  your  analyses . 

0      If  relevant,  discuss  your  project's  cross-sectional  aspects 
(comparisons  in  one  time  period)  and  longitudinal  aspects 
(comparisons  over  time) . 

•  Avoid  left-overs!     It  should  be  clear  by  the  end  of  this 
section  that  you  will  not  collect  data  for  which  you  have  no 
obvious  use  in  this  study  and  that  you  will  have  obtained 
pertinent  data  for  all  the  topics  you  propose  to  address. 

Numerous  issues  arise  with  respect  to  data  collection  instruments 
and  people.     If,  for  instance,  you  are  using  or  adapting  existing  self- 
reported  measures  such  as  questionnaires  that  patients  or  care-givers 
complete,  can  you  document  that  they  are  known  to  be  reliable?  (That 
is,  is  there  evidence  that  they  will  assess  patient  characteristics  or 
attributes  the  same  way  if  used  more  than  once  or  if  administered  in 
different  ways?)    Further,  can  you  document  that  they  are  known  to  be 
valid?     (That  is,  do  they  really  assess  the  factors  or  variables  they 
purport  to  assess  and  not  something  else?) 

If  your  data  collection  instruments  already  exist  in  some  form, 
consider  including  them  (or  at  least  a  subset)  as  an  appendix.     If  you 
are  going  to  get  help  from  persons  knowledgeable  about  these 
instruments,  such  as  the  original  developers,  say  so. 

If  you  are  developing  your  own  measures  or  instruments,  indicate 
how  you  will  establish  their  reliability  and  validity.     In  this 
instance,  you  should  give  at  least  some  idea  of  what  such  forms  might 
look  like  or  what  elements  (e.g.,  individual  illustrative  questions) 
they  might  contain.     As  with  most  research  projects,  developing  data 
collection  instruments  usually  does  not  start  completely  from  scratch, 
so  if  you  have  examples  of  materials  on  which  you  might  base  your  own 
forms,  include  them  and  indicate  how  you  expect  to  modify,  update,  or 
improve  them. 
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If  you  are  using  interviewers,  medical  record  abstractors,  or  other 
data  collection  personnel,  you  should  give  some  description  of  how  they 
will  be  selected  and  trained.     How  will  you  ensure  that  they  all  do 
things  in  pretty  much  the  same  way,  so  that  in  the  end  you  have 
comparable  data  across  all  your  study  participants? 

Your  will  probably  need  to  get  informed  consent  from  potential 
subjects  to  participate  in  your  demonstration,  unless  you  will  be 
collecting  only  anonymous  data.     This  section  is  as  good  a  place  as  any 
to  describe  how  you  will  carry  out  the  informed  consent  procedures.2 

Finally,  this  section  would  be  a  good  place  to  distinguish  between 
two  types  of  data  you  may  collect  in  your  study:     primary  (gathered 
directly  by  you  from  your  subjects)  and  secondary  (drawn  from  sources 
external  to  your  own  direct  data-gathering) .     Secondary  data  are  often 
useful,  but  their  value  may  be  limited  by  a  lack  of  comparability 
between  your  own  subjects  and  those  on  which  the  secondary  data  were 
based.     Thus,  if  you  plan  to  draw  on  secondary  data  sources  you  should 
discuss  both  their  advantages  and  limitations  for  your  project. 

Data  collection  problems.     If  you  foresee  special  data  collection 
problems,  indicate  what  they  are  and  how  you  will  try  to  overcome  them. 
For  instance,  might  you  have  difficulty  gaining  access  to  medical 
records  in  local  hospitals  or  physicians'  offices,  or  in  obtaining 
information  on  the  total  amounts  of  medical  services  study  participants 
use  during  the  project?    Or,  what  will  you  do  if  certain  individuals,  or 
types  of  study  participants,  do  not  respond  to  your  surveys?     It  is 
better  to  show  that  you  know  where  the  problems  are  and  that  you  have 
given  them  some  thought  than  to  have  reviewers  assume  that  you  were  not 
aware  difficulties  might  arise.3 

2 Informed  consent  is  part  of  a  larger  concern  for  protecting  human 
subjects --that  is,  what  steps  have  been  taken,  such  as  getting  approval 
from  an  Institutional  Review  Board,  to  safeguard  the  rights  and  welfare 
of  your  study  subjects.     This  is  typically  done  on  a  separate  HCFA  or 
DHHS  form  (see  the  application  kit).     Explicit  requirements  can  be  found 
in  the  Code  of  Federal  Regulations,  Title  45  on  Public  Welfare,  Section 
46  on  Protection  of  Human  Subjects.     The  Office  of  Protection  from 
Research  Risks  of  the  National  Institutes  of  Health  in  Bethesda, 
Maryland,  publishes  an  "OPRR  Report"  containing  the  relevant 
regulations . 

3 If  you  are  doing  a  project  for  which  federal  funding  is  requested 
(a  project  that  is  not  a  waiver-only  demonstration),  or  if  you  expect 
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Data  base  management.    The  larger  your  proposed  project,  the 
greater  the  amount  of  information  you  will  amass.     Regardless  of  the 
size  of  your  project,  however,  the  quantity  of  data  you  eventually 
collect  will  probably  be  larger  than  you  expect.    Thus,  no  matter  how 
large  your  study,  you  should  comment  explicitly  on  how  your  data  will  be 
held,  managed,  and  processed. 

For  instance,  who  will  have  the  main  responsibility  for  organizing, 
storing,  and  archiving  completed  questionnaires?    Who  will  maintain 
computer  data  tapes  and  make  needed  workfiles  available  to  those  who 
will  analyze  the  data?    How  will  the  privacy  of  information  on  study 
participants  be  guarded  and  guaranteed? 

D.    Evaluation  and  Analysis  Plan 
D.1.    General  Points 

In  an  ODE  demonstration  project,  the  plans  you  have  for  analyzing 
your  data  form  your  evaluation  plan,  and  they  should  be  discussed  here. 
The  reasoning  is  essentially  as  follows:     You  are  mounting  a 
demonstration  to  explore  the  feasibility  and  practicality  of  some 
program  or  to  test  some  hypotheses  about  that  program,  be  it  a  new 
service  to  Medicare  beneficiaries  or  a  different  way  of  making  care 
available  to  mothers  or  children  eligible  for  Medicaid.    To  know  if  your 
demonstration  has  been  successful,  you  will  have  to  evaluate  it.*  That 
is,  you  will  have  to  carry  out  a  series  of  specific  analysis  tasks, 
either  throughout  the  demonstration  or  near  its  conclusion,  that  will 
permit  you  to  conclude  something  about  the  feasibility  and  possible 
general izability  of  the  program  or  about  the  degree  to  which  your 
hypotheses  have  been  supported  or  rejected. 

HCFA  to  specify  the  data  collection  instrument,  then  a  question  of 
clearance  may  arise.     Specifically,  in  those  situations,  if  you  are 
planning  on  a  survey  of  more  than  nine  respondents,  you  may  need  to  get 
clearances  and  approvals  of  your  survey  instruments  and  procedures  from 
the  Federal  Office  of  Management  and  Budget  (OMB).     Hence,  you  may  need 
to  make  sure  your  overall  study  design  has  allowed  for  the  appropriate 
activities  and  time  (often  four  to  six  months)  for  this. 

''In  funding  demonstration  projects,  ODE  assumes  that  the  analysis 
or  evaluation  plan  you  provide  in  the  proposal  could  be  done  by  you. 
However,  the  agency  does  reserve  the  right  to  pull  out  the  analysis  plan 
and  have  it  done  by  others . 


-  14  - 


The  elements  of  an  evaluation  plan  differ  according  to  the  three 
categories  of  demonstrations.     In  Category  I  projects,  the  evaluation 
plan  is  built  directly  into  your  experimental  research  design,  so  in 
some  sense  a  separate  plan  by  which  the  project  can  be  evaluated  is 
unnecessary.     You  will  discuss  what  parts  of  the  analysis  are  directed 
at  each  hypothesis  stated  at  the  outset. 

For  Category  II  projects,  you  are  required  to  give  a  separate  plan 
by  which  your  project  can  be  evaluated.     It  should  cover  in  detail  what 
aspects  of  the  demonstration  need  to  be  evaluated,  how  you  propose  to 
judge  them,  and  by  what  criteria.     You  should  be  trying  to  answer  the 
questions:     "How  well  did  what  we  tried  to  do  work?"  and  "What  did  we 
learn  from  the  project?"    Finally,  for  Category  III  projects  you  should 
discuss  the  relevant  evaluation  issues  as  noted  for  Category  II  and 
describe  a  general  framework  for  evaluation.     Category  III  projects  are 
likely  to  be  less  complex  than  the  other  types  of  projects,  but  because 
they  may  have  a  weaker  design  (experimentally  speaking)  it  is  all  the 
more  important  that  the  analysis  be  well  thought  out. 

D.2.    Analysis  Plan 

This  is  the  section  where  you  explain,  as  clearly  as  possible,  how 
you  will  use  the  data  you  have  collected.     These  data  will  presumably 
allow  you  to  test  the  hypotheses  you  set  out  earlier,  to  examine  how 
practical  your  program  is,  or  to  determine  how  applicable  your  services 
might  be  for  wider  populations.     This  section  should  convince  reviewers 
that  your  methods  are  consistent  with  the  questions  you  have  asked  and 
the  data  you  have  collected,  and  it  should  persuade  them  that  the 
quality  and  nature  of  the  data  will  support  the  level  of  analysis  you 
have  in  mind. 

Analytic  methods.     In  particular,  you  should  indicate  here  what 
analytic  methods  you  expect  to  use  to  address  which  questions.  For 
instance,  you  may  want  to  rely  on  detailed  narrative  descriptions  to 
show  how  the  implementation  of  your  program  in  several  sites  was  similar 
or  different  across  those  sites.     Your  reviewers  may  then  look  for  some 
indication  or  listing  of  the  topics  you  expect  to  discuss  in  that 
narrative. 
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For  other  parts  of  the  analysis,  however,  you  may  plan  to  employ 
such  statistical  techniques  as  multi-level  contingency  tables,  analysis 
of  variance,  and/or  linear  regression.     Reviewers  may  want  to  see  some 
general  statement  (a  model)  of  what  variables  you  expect  to  be 
associated  with  the  program  outcomes  you  are  studying;  you  might  display 
this  as  an  equation  or  as  a  flow  diagram.     For  these  more  technical 
analyses,  you  should  give  some  idea  of  the  tests  you  will  apply  to  see 
if  differences  between  groups  or  changes  over  time  in  program  outcomes 
are  statistically  significant. 

It  is  often  helpful  to  give  examples  of  your  analyses  or  to  show 
what  you  think  your  final  tables  of  results  might  look  like.  Often, 
discussing  hypothetical  findings  based  on  likely  values  of  the  data  you 
will  eventually  collect  is  a  useful  device  for  making  the  analysis  plan 
seem  less  abstract.     In  essence,  you  are  trying  to  aid  reviewers  in 
visualizing  the  data  set  you  will  compile,  so  tftat  they  can  think  along 
with  you  about  what  methods  of  analyses  seem  appropriate  and  reasonable 
to  answer  the  questions  you  have  set  out  to  answer. 

Analytic  pitfalls.    As  with  data  collection  efforts,  it  is  better  to 
acknowledge  possible  problems  with  your  analysis  and  conclusions  drawn 
from  it,  and  to  indicate  how  you  would  overcome  those  that  seem  most 
troublesome,  than  to  ignore  such  issues  altogether.     For  instance,  will 
you  run  the  risk  of  a  Hawthorne  effect,  in  which  special  attention  to 
some  study  subjects  may  either  mask  or  enhance  the  effects  you  are 
actually  trying  to  measure?    An  example  of  such  an  effect  would  be  doing 
several  interviews  of  elderly  participants  who  are  otherwise  relatively 
isolated;  just  the  attention  being  paid  to  them  may  have  some 
unanticipated  influence  on  their  responses  or  activities.     As  another 
example  of  analytic  pitfalls,  if  you  are  going  to  identify  for 
additional  or  repeat  analyses  some  special  group,  who  may  score  high  or 
low  on  some  measure  of  interest,  will  you  be  able  to  account  for  the 
possible  regression  to  the  mean  phenomenon?    This  says  simply  that,  just 
by  chance  alone,  people  who  are  very  high  (low)  at  one  time  are  likely 
to  go  down  (up)  the  next  time  they  are  studied. 
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There  are  many  other  standard  analytic  pitfalls,  some  of  which  will 
be  relevant  to  your  study  and  some  of  which  will  not  be;  you  may  already 
know  about  many  of  them,  and  you  can  examine  some  of  the  books  in  the 
bibliography  for  details  about  others.     It  is  a  good  idea  to  consult  a 
statistician,  econometrician,  or  some  other  person  well  acquainted  with 
basic  research  methodology  when  you  are  planning  the  design  and  analysis 
of  your  demonstration;  your  proposal  will  likely  be  strengthened 
substantially  by  discussions  with  such  people. 

D.  3.    Phase-Down  Plan 

All  demonstration  proposals  must  include  a  section  that  describes 
how  your  project  will  wind  down,  and  this  can  be  discussed  as  part  of 
the  evaluation  plan.    That  is,  you  need  to  say  how  you  will  make  a 
smooth  transition  from  the  end  of  a  demonstration  to  whatever  would  come 
next  (typically,  no  longer  giving  the  services  directly  through  the 
project).     You  should  indicate  how  and  when  you  will  inform  program 
beneficiaries  that  the  project  is  coming  to  an  end;  for  instance,  you 
may  plan  to  tell  your  participants  three  months  ahead  of  time  ".   .  .to 
remember  that  starting  on  such-and-such  a  date,  we  will  no  longer  be 
providing  such-and-such  services."    Although  this  may  be  of  greater 
importance  to  ODE  staff  than  to  your  peer  reviewers,  the  latter  will 
appreciate  your  attempt  to  minimize  disruption  in  your  study 
participants'  lives. 

E.    Work  PSan 

E.  1.    Description  of  Tasks 

Your  work  should  be  sufficiently  well  planned  that  you  can  specify 
a  set  of  tasks  that  will  cover  all  the  activities  needed  to  complete  the 
project.    There  are  no  hard-and-fast  rules  here,  although  a  general 
guideline  might  be  that  a  multi-year  Category  I  proposal  could  have  30 
or  so  discrete  tasks  to  describe.     For  instance,  assume  that  part  of 
your  work  calls  for  you  (1)  to  adapt  an  existing  patient  questionnaire 
on  health  status  for  use  in  your  demonstration,   (2)  to  administer  it 
once  as  you  enroll  people  into  the  demonstration,  and  (3)  to  administer 
it  again  at  the  end.     Then  you  probably  have  (at  least)  three  separate 
tasks  for  which  to  account.     It  helps  to  number  the  tasks. 
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The  aim  is  to  cover  all  the  things  you  promised  to  do  in  Sections  C 
and  D  regarding  study  design  and  analysis.     In  addition,  note  that  one 
task  will  probably  involve  producing  a  final  report. 

This  is  a  good  spot  to  check  that  you  have  no  loose  ends.  Every 
task  noted  here  should  have  some  corresponding  description  in  the 
methods  to  show  how  it  will  be  accomplished;  every  major  activity  you 
promised  to  do  should  have  a  corresponding  task. 

E.2.    Time  Schedule 

You  should  provide  a  Gantt  chart  or  some  other  diagram  to 
illustrate  when  the  tasks  just  outlined  will  be  done,  in  what  order,  and 
how  long  they  are  expected  to  take.    This  is  commonly  done  in  terms  of 
elapsed  months  (e.g.,  for  a  two-year  study,  months  0  through  24  would  be 
one  axis  of  your  chart),  but  if  some  other  timeframe  seems  more 
appropriate  for  you,  use  it.     It  is  helpful  to  adopt  some  conventional 
symbols,  such  as  an  asterisk  or  triangle,  to  show  when  specific 
milestones  are  to  be  achieved. 

Working  out  the  time  schedule  may  seem  burdensome,  but  it  helps  you 
avoid  awkward  problems  that  your  reviewers  may  well  detect.  For 
example,  you  do  not  want  to  say  that  you  will  follow  your  study 
participants  for  three  years  and  analyze  data  on  all  participants,  and 
then  show  that  data  analysis  is  over  at  the  end  of  the  third  year. 

E.3.    Level  of  Effort  of  Personnel 

This  section  is  commonly  shown  as  a  table,  in  which  you  list  the 
key  individuals  (by  name  or  by  role  in  the  demonstration)  and  the  number 
of  days  they  will  devote  to  each  task.    That  is,  what  you  expect  your 
personnel  to  do  should  be  tied  to  the  tasks  you  previously  described. 

For  multi-year  demonstrations,  you  should  show  total  days  in  each 
year.  Total  days  per  year  should  be  equivalent  to  whatever  percentage 
of  time  you  show  for  these  individuals  in  your  budget  document. 

This  is  one  more  chance  to  make  sure  that  what  you  propose  to  do 
looks  possible.     Reviewers  pay  attention  to  these  figures.     Too  little 
time  for  key  personnel  suggests  you  may  have  an  unrealistically 
optimistic  view  of  what  you  can  accomplish. 
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F.    Project  Staff 

F.I.    Qualifications  of  Key  Staff 

To  the  extent  possible,  persons  you  believe  are  crucial  to  a 
successful  demonstration  project  should  be  known  to  you  and  named  in 
this  section.     Even  very  good  projects  will  look  dubious  to  reviewers  if 
the  principal  investigator  or  critical  staff  are  "to  be  named." 

You  should  briefly  discuss  the  qualifications  of  key  personnel 
named  in  this  section.     Probably  a  paragraph  or  two  per  person  will  do, 
as  you  need  indicate  only  what  in  their  background  and  experience  is 
most  pertinent  to  this  demonstration.     (You  will  append  to  the  proposal 
full  curricula  vitae  on  all  these  individuals.) 

This  or  a  parallel  section  could  also  be  used  to  describe  any 
experience  your  organization  has  had  in  conducting  similar 
demonstrations,  especially  insofar  as  that  experience  will  be  available 
as  backup  and  support  for  the  key  staff. 

If  you  have  special  data  collection  or  analytic  needs,  this  is  the 
place  to  indicate  that  you  have  the  right  personnel  for  the  job.  Often, 
these  individuals  can  be  consultants  rather  than  project  staff.  For 
instance,  you  may  need  a  physician  or  psychologist  for  certain  tasks  and 
a  statistician  or  economist  for  other  tasks.     To  the  degree  you  can, 
indicate  who  these  people  are  or  say  what  types  of  individuals  you  will 
later  recruit. 

Subcontracting  for  very  specialized  work,  such  as  abstracting 
medical  records  or  conducting  a  survey,  may  be  an  option  for  you.  In 
these  instances,  if  the  subcontractor  arrangement  has  not  already  been 
settled,  you  should  be  explicit  about  whom  you  have  in  mind  or  what 
criteria  you  would  use  to  select  a  subcontractor. 

F.2.    Organizational  Chart 

In  a  complex  project,  you  should  state  who  is  responsible  for  what 
sets  of  activities  and  how  those  individuals  relate  to  one  another  and 
to  the  principal  investigator  and/or  project  director.     For  multi-site 
projects,  you  should  also  say  who  acts  as  the  liaison  across  the  sites. 
For  projects  involving  subcontractors  (such  as  the  organization  that 
does  just  the  survey  work  or  provides  the  particular  services),  you 
should  show  which  individual (s)  are  responsible  for  those 
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subcontractors.     It  should  be  possible  to  indicate  all  this  in  a  single 
organizational  chart. 

G.    Implementation  Potential 

This  is  not  a  long  section,  typically,  but  it  is  an  important  one. 
It  is  where  you  discuss  the  expected  use,  generalizability , 
applicability,  and  dissemination  of  your  work.    That  is,  what  do  you  see 
as  the  likely  application  or  dispersion  of  the  program  or  innovation  you 
have  been  studying  to  other  settings  or  facilities,  or  to  other  parts  of 
the  country,  or  by  other  researchers? 

If  you  can  foresee  possible  obstacles  to  implementation  of  your 
work  elsewhere,  you  probably  should  mention  them;  reviewers  will  see 
them,  too.     Indicate  why  they  may  be  less  troublesome  than  customarily 
expected  or  how  they  might  be  gotten  around.     Conversely,  if  there  are 
factors  that  facilitate  implementation,  note  them  also. 

Other  Parts  of  the  Project  Narrative 

By  now  you  may  have  a  long,  but  one  hopes  not  overly  long, 
proposal,  and  the  question  arises  as  to  what  else  you  might  do  to  make 
the  proposal  clearer  and  to  ease  the  reviewers'  burden.     Remember  that 
some  reviewers  will  be  traveling  to  the  peer  review  panel  meeting, 
carrying  numerous  and  lengthy  documents  with  them! 

First,  a  Table  of  Contents  for  the  Project  Narrative  section 
(including  its  appendices)  is  a  big  favor  to  your  reviewers. 
Implicitly,  then,  it  is  helpful  to  everyone  to  number  your  pages. 
Second,  successful  proposals  often  contain  an  Executive  Summary,  which 
should  be  short  and  yet  should  cover  the  critical  points  of  what  you  aim 
to  do.     That  is,  it  would  summarize  Parts  A  through  D  above. 

Third,  examples  of  data  collection  instruments  and  letters  of 
support  and  commitment  from  professional  organizations,  local  health 
facilities,  or  possible  consultants  can  all  be  included  as  appendices . 
Be  sure  that  appendix  designations  (by  letter  or  numeral)  correspond  to 
those  you  have  used  in  the  Project  Narrative  text.     Technical  appendices 
on  design  or  sampling  issues,  such  as  any  power  calculations  you  have 
done,  can  be  helpful;  reviewers  who  want  to  go  over  such  details  can  do 
so  and  reviewers  who  will  not  check  such  materials  can  ignore  the 
appendix. 
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You  should  probably  forgo  putting  some  things  into  any  appendix. 
These  include  reprints  of  other  work  you  have  done  and  reprints  of 
articles  that  other  investigators  have  written.     Presumably  you  have 
covered  this  material  and  experience  in  the  literature  review,  so  unless 
such  information  is  critical  to  understanding  your  present  demonstration 
or  substitutes  for  a  technical  appendix,  leave  it  out. 

Finally,  make  sure  your  references  are  complete,  accurate,  and 
match  what  you  have  cited  throughout  the  entire  document.  For 
publications  cited  in  technical  appendices,  you  can  either  have  a 
separate  reference  list  directly  following  the  appendix  or  incorporate 
the  citations  into  your  main  list. 

V.    FINAL  POINTS 

If  you  are  uncomfortable,  at  this  stage  of  reading  this  brief 
guide,  with  any  aspects  of  the  proposal  you  intend  to  write,  get  help. 
For  instance,  as  mentioned  in  the  analysis  plan  section  above, 
successful  applicants  often  call  on  statistical  consultants  (perhaps 
from  a  local  university)  for  assistance  during  the  proposal-writing 
stage.     Such  experts  may  also  be  involved  for  a  portion  of  their  time 
throughout  your  project.     You  may  also  wish  to  contact  ORD  if  you  have 
specific  questions. 

Successful  proposals  also  seem  to  benefit  from  a  few  other 
techniques.     Perhaps  the  least  obvious,  but  often  most  rewarding,  is  to 
ask  someone  not  on  the  proposal-writing  team  to  read  through  a  draft  and 
tell  you  where  the  problems  are! 

Use  enough  "road  signs"  to  help  your  reviewers  understand  where  you 
are  heading  and  how  you  got  there.     Subheadings  are  important;  you  might 
try  an  outline  numbering  system  for  all  levels  of  headings  (remembering 
that  the  Project  Narrative  is  Part  IV).     Long,  dense  pages  of  text  are 
daunting,  so  divide  lengthy  paragraphs  into  smaller  ones. 

We  have  concentrated  on  the  Project  Narrative  section  of  the 
proposal,  but  one  point  about  budget  remains.     Make  sure  that  your 
budget  matches  up  with  tasks  and  personnel  named  in  Sections  E  and  F 
above.    For  instance,  are  the  key  people  named  above  covered  in  the 
budget  for  the  time  you  said  it  would  take  them  to  do  all  their  assigned 
tasks? 
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Your  proposal  does  not  have  to  be  fancy  in  appearance,  but  it 
should  be  neat  and  attractive.     Check  for  consistency,  spelling,  and 
other  such  details  enough  ahead  of  time  that  you  can  make  corrections. 
Try  to  write  in  as  lively  and  active  a  way  as  possible;  it  will  shorten 
your  proposal  and  convey  enthusiasm  and  confidence  about  your  proposed 
work.    With  those  attributes,  a  clear  statement  of  objectives,  a 
consistent  plan  for  data  collection  and  analysis,  and  appropriate 
personnel,  you  are  well  on  your  way  to  a  proposal  reviewers  will  applaud 
and  recommend  for  approval . 
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SELECTED  GLOSSARY  OF  RESEARCH  AND  DEMONSTRATION  TERMS 


Bias:     The  systematic  tendency  to  over-  or  underestimate  the  true  value 
of  some  quantity  of  interest,  often  arising  from  the  use  of 
nonrepresentative  sampling  schemes  or  from  a  reliance  on  noncomparable 
treatment  and  control  groups  (see  comparison  group).    For  example, 
using  55-  to  64-year-olds  in  a  control  group  of  a  study  of  the  impact 
of  the  Prospective  Payment  System  (PPS)  on  quality  of  health  care  for 
65-and-over  Medicare  patients  would  certainly  bias  the  results  against 
the  treatment  (PPS)  unless  steps  were  taken  in  the  analysis  to  adjust 
for  prior  health  status  (a  potential  confounding  factor);  the  first 
group  will  be  systematically  healthier  than  those  in  the  PPS  group, 
just  because  they  are  younger  and  will  therefore  exhibit  fewer 
negative  outcomes  (deaths,  hospital  readmissions ,  and  so  on)  than  the 
treatment  group,  even  if  PPS  has  no  effect. 

Comparison  group:    The  point  of  most  demonstrations  is  to  estimate  the 
causal  effect  of  a  program,  or  treatment,  on  one  or  more  outcomes,  or 
responses,  or  dependent  variables  of  interest.     In  the  PPS  example 
above,  the  treatment  is  receiving  care  under  PPS,  and  the  outcomes  are 
measures  of  the  quality  of  health  care  such  as  rates  of  hospital 
readmission  and  postdischarge  mortality.     The  goal  is  to  estimate  the 
difference  between  how  things  came  out  for  a  group  of  people  who 
receive  the  treatment  (the  treatment  group  or  experimental  group) 
and  how  things  would  have  come  out  for  those  people  if  they  had  not 
received  the  treatment .     Sometimes  it  is  possible  to  get  a  fairly 
direct  idea  of  the  latter  by  measuring  the  same  people  twice,  once 
before  they  receive  the  treatment  and  then  again  afterward;  this  is  an 
example  of  a  pre-post  (before-after)  design,  in  which  subjects  serve 
as  their  own  controls,  and  the  analysis  focuses  on  the  differences  for 
each  subject  between  the  pre  and  the  post  values  of  the  outcomes  (a 
paired  comparison) .     But  sometimes  this  is  not  possible;  then  it  is 
necessary  to  use  information  from  a  second  group  of  people  who  do  not 
receive  the  treatment,  the  comparison  or  control  group,  as  a  proxy 
for  how  the  treated  people  would  have  responded  if  they  had  not  been 
given  the  treatment.     The  analysis  is  then  based  on  the  differences  in 
the  outcomes  of  interest  between  treatment  and  control  subjects  (a  two- 
sample  comparison) . 

Control  group:    See  comparison  group. 

Controlled  experiment:     An  experiment  conducted  under  conditions  in 
which  the  experimenters  exercise  control  over  who  goes  into  the 
treatment  and  comparison  groups,  as  contrasted  with  an  observational 
study,  in  which  no  such  control  is  exercised.     A  study  on  the 
efficacy  of  home  health  care  in  preventing  institutionalization  among 
Medicare  patients  could  be  conducted  as  a  controlled  experiment,  since 
presumably  it  would  be  possible  to  assign  eligible  Medicare  patients 
to  a  group  receiving  home  health  aide  intervention  or  to  a  group  not 
receiving  such  care.     However,  a  study  looking  into  the  relationship 
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between  smoking  and  lung  cancer  would  have  to  be  observational, 
because  ethically  no  one  can  be  forced  to  smoke  just  to  find  out  if 
his  or  her  chance  of  getting  lung  cancer  would  increase. 

Covariate:    See  potential  confounding  factor. 

Dependent  variable:    See  outcome  variable  and  comparison  group. 
Experimental  group:    See  comparison  group. 

Independent  variable:    A  variable  that  is  useful  in  predicting  the 
values  of  the  outcome  measures,  also  called  an  experimental  or 
predictor  variable.     Typically  these  variables  are  under  direct 
control  of  the  researcher,  as  for  instance  the  treatment -control 
distinction  for  each  subject  in  a  controlled  experiment.  Variables 
that  are  useful  predictors  but  are  not  under  experimenter  control  are 
often  referred  to  as  covariates  or  potential  confounding  factors. 

Informed  consent:    The  process  by  which  you  will  fully  explain  to 
potential  enrollees  or  participants  (or  their  proxies)  the  study's 
purposes  and  activities,  and  the  rights  and  obligations  of  both 
participants  and  investigators,  and  obtain  their  agreement  to 
participate.     It  is  in  this  context,  for  instance,  that  you  will 
pledge  to  protect  the  privacy  and  confidentiality  of  any  information 
you  may  obtain  from  participants. 

Observational  study:    See  controlled  experiment. 

Outcome  variable(s):    One  or  more  variables  that  are  of  principal 
interest  or  value  in  assessing  the  effect  of  a  program  or  treatment; 
see  comparison  group. 

Population:     The  collection  of  people  or  things  to  which  you  wish  to 
generalize  on  the  basis  of  your  sample. 

Potential  confounding  factor:     Demonstrating  that  treatment  and  control 
groups  differ  noticeably  on  the  outcome  variables  (in  other  words, 
showing  an  association  between  the  treatment  and  response  variables) 
is  not  enough  to  prove  that  the  treatment  caused  the  difference 
observed.     The  two  groups  may  differ  in  other  important  ways  than  just 
the  treatment-control  distinction,  and  until  shown  otherwise  these 
sources  of  noncomparability  could  cause  the  outcome  differences  just 
as  well  as  the  treatment  could.     If  a  twofold  argument  can  be  made 
about  a  variable—namely ,  that  the  variable  is  likely  to  take  on 
systematically  different  values  in  the  treatment  and  control  groups, 
and  that  the  outcome  measure  is  associated  with  the  variable- -then  the 
variable  is  called  a  potential  confounding  factor.     In  a  study 
relating  use  of  the  contraceptive  pill  to  hypertension  in  women 
between  18  and  45,  for  instance,  age  would  be  a  clear  potential 
confounding  factor,  since  older  women  use  the  pill  less  and  also  have 
higher  blood  pressure  on  average.    When  noncomparabilities  between 
treatment  and  control  groups  are  evident  at  analysis  time,  it  is  often 
possible  to  reduce  the  bias  arising  from  such  noncomparabilities  by 
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means  of  a  modeling  technique  known  as  analysis  of  covariance  or 
covariate  ad  just  merit .     Thus,  potential  confounding  factors  are  also 
sometimes  referred  to  as  covariates. 

Power:    The  chance  of  finding  a  difference  in  the  outcome  measures 
between  the  treatment  and  control  people,  if  a  difference  is  there  to 
be  found.     The  power  of  a  statistical  procedure  to  find  differences 
depends  on  several  things,  mainly  the  size  of  the  difference  (bigger 
differences  are  easier  to  find)  and  the  sample  size  (smaller 
differences  can  be  found  with  more  data).     It  is  important  at  the  time 
an  experiment  is  designed  to  make  sure  that  its  resources  are  well 
matched  to  its  needs --that  is,  to  go  through  a  process  of  sample  size 
determination .     This  usually  requires  one  or  more  power  calculations . 
In  calculations  of  this  type,  some  thought  is  put  into  what  magnitudes 
of  differences  on  the  outcome  measures  are  worth  detecting  from  a 
policy  point  of  view;  then,  the  sample  sizes  needed  to  have  a 
reasonable  chance  of  finding  differences  of  that  size  (if  they  are 
there  to  be  found)  are  computed. 

Randomization:    The  standard  way  of  achieving  at  least  approximate 
comparability  of  comparison  groups  in  controlled  experiments  is  to 
assign  subjects  to  treatment  and  control  at  random.    This  ensures 
exact  comparability  with  respect  to  all  potential  confounding  factors 
in  a  long -run -average  sense;  on  average,  across  independent 
replications  of  the  experiment,  all  the  ways  in  which  the  treatment 
and  control  subjects  might  be  different  will  balance  out. 
Randomization  cannot  be  relied  on  to  achieve  exact  comparability  each 
time  it  is  employed,  but  particularly  with  fairly  large  single  sites 
it  can  be  expected  to  give  rough  comparability  on  most  or  all 
variables  in  any  given  performance  of  the  experiment. 

Reliability:    The  extent  to  which  a  measuring  instrument  like  a  survey 
produces  the  same  answers  when  given  to  the  same  subjects  under 
somewhat  different  conditions,  for  instance  when  administered  by 
different  people  or  when  presented  to  the  same  people  at  two  points  in 
time.    The  issue  being  addressed  by  the  question  "How  reliable  is  this 
survey?"  is  essentially  the  following:     How  much  do  extraneous  factors 
(such  as  who  administers  the  instrument,  or  how  the  person  filling  out 
the  questionnaire  happens  to  feel  at  the  time)  affect  the  results  of 
the  survey?    As  contrasted  with  validity. 

Response  variable:    See  outcome  variable  and  comparison  group. 

Sample:     The  subset  of  the  population  of  interest  on  which  your 
inference  is  to  be  based;  see  population. 

Treatment  effect:     Another  name  for  the  (adjusted  or  unadjusted)  average 
difference  on  the  outcome  variable  scale  between  the  pre  and  post 
values  in  a  pre-post  design  or  between  the  treatment  and  control 
values  in  a  two-sample  comparison;  see  comparison  group. 
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Treatment  variable:    See  independent  variable. 

Validity:     The  extent  to  which  a  measuring  instrument,  such  as  a  battery 
of  questions  in  a  survey,  actually  measures  what  it  purports  to 
measure.     Do  you  have  some  items  that  you  think  measure  aptitude,  when 
in  fact  what  they  really  measure  is  achievement?     If  so,  you  have  a 
validity  problem.    As  contrasted  with  reliability. 

Waiver:     As  used  by  HCFA/ORD,  approval  by  HCFA  to  deviate  from  the 
requirements  of  Title  XVIII  or  Title  XIX  of  the  Social  Security 
Amendments  (Medicare  and  Medicaid,  respectively)  and  their 
implementing  regulations,  to  conduct  a  time-limited  demonstration. 
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